Multimodal Input for Perceptual User Interfaces

Ever since Bolt’s seminal paper, ”Put that there: Voice and Gesture at the Graphics Interface”, the notion that multiple modes of input could be used to interact with computer applications has been an active area of human computer interaction research (Bolt 1980). This combiniation of different forms of input (e.g., speech, gesture, touch, eye gaze) is known as multimodal interaction and its goal is to support natural user experiences by providing the user with choices in how they can interact with a computer. These choices can help to simplify the interface, provide more robust input when recognition technology is used, and support more realistic interaction scenarios because the interface can be more fine tuned to the human communication system. More formally, multimodal interfaces process two or more input modes in a coordinated manner which aim to recognize natural forms of human language and behavior and typically incorporate more than one recognition-based technology (Oviatt 2003). With the advent of more powerful perceptual computing technologies, multimodal interfaces that can passively sense what the user is doing are becoming more prominent. These interfaces, also called perceptual user interfaces (Turk and Robertson 2000), provide mechanisms that support unobtrusive interaction where sensors are placed in the physical environment and not on the user. The prior chapters in this book have focused on various input technologies and associated interaction modalities. In this chapter, we will examine how these different technologies and their input modalities, specifically speech, gesture, touch, eye gaze, facial expressions, and brain input can be combined and the types of interactions they afford. We will also examine the strategies for combining these input modes together, otherwise known as multimodal integration or fusion. Finally, we will examine some usability issues with mutlimodal interfaces and methods for handling them. Research in multimodal interfaces spans many fields including psychology, cognitive science, software engineering,